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s Кодифицированные знания 


ClickHouse Operator creates, configures and manages ClickHouse clusters running on Kubernetes. 


release v0.14.0 | © PASSED | docker pulls 356k ] go report A+ 80 0 


Features 


The ClickHouse Operator for Kubernetes currently provides the following: 


e Creates ClickHouse clusters based on Custom Resource specification provided 
e Customized storage provisioning (VolumeClaim templates) 

e Customized pod templates 

e Customized service templates for endpoints 

e ClickHouse configuration and settings (including Zookeeper integration) 

e Flexible templating 

e ClickHouse cluster scaling including automatic schema propagation 

e ClickHouse version upgrades 


e Exporting ClickHouse metrics to Prometheus 
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https://github.com/Altinity/clickhouse-operator 


| ClickHouse 


Altinity Ke 


ALTINITY.CLOUD 


Fully Managed ClickHouse for the Amazon Cloud 
by the Enterprise Experts 


Create Prod-Ready Clusters in Any AWS 
Region with Ease 


Spin up clusters, connect, and start working. Vertical and horizontal scaling? 
The Altinity.Cloud cluster manager makes it a snap. Zookeeper? You'll forget 
it's there. High availability? Altinity.Cloud has multi-AZ operation and 
automatic backup. Upgrade? Altinity.Cloud does it automatically without 
interrupting service. Monitoring? Dashboards are built-in and ready to use. 


Altinity.Cloud runs in the region your applications need, not the region that's 
convenient for us. Pick any Amazon region and we'll make it work. Active 
regions are available for immediate deployment. We can add new regions in 
couple of days, so don't be shy. 
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ClickHouse Operator creates, configures and manages ClickHouse clusters running on Kubernetes. 
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Features 


The ClickHouse Operator for Kubernetes currently provides the following: 


e Creates ClickHouse clusters based on Custom Resource specification provided 
e Customized storage provisioning (VolumeClaim templates) 

e Customized pod templates 

e Customized service templates for endpoints 

e ClickHouse configuration and settings (including Zookeeper integration) 

e Flexible templating 

e ClickHouse cluster scaling including automatic schema propagation 

e ClickHouse version upgrades 


e Exporting ClickHouse metrics to Prometheus 
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Features 


The ClickHouse Operator for Kubernetes currently provides the following: 


e Creates ClickHouse clusters based on Custom Resource specification provided 
e Customized pod templates 

e Customized service templates for endpoints 

e ClickHouse configuration and settings (including Zookeeper integration) 

e Flexible templating 

e ClickHouse cluster scaling including automatic schema propagation 

e ClickHouse version upgrades 


e Exporting ClickHouse metrics to Prometheus (нг) High. oad 
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Persistent Storage 


° Cloud storage 
o AWS • CSI 


o GKE 


o Other cloud providers os 
* Provisioners 


s Local storage 
o emptyDir 
o hostPath 
o Local 
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Рассмотрим 
через призму 
users & изе-сазез 
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Level: 
Начальный. Cloud storage 


Описание: 
Случайно поменял одну 
строчку, и все сломалось! 


Запрос: 
Верните все назад! da 
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Persistent Storage — cloud storage 
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apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: clusterl 
- name: cluster2 
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apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 

name: "simple-02" 
spec: 

configuration: 

clusters: 
- name: clusterl 
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apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: clusterl 
- name: cluster2 


Г\ HighLoad 
(HL) м dd 


apiVersion: "clickhouse.altinity 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: "shardl-repll" 
layout: 
shardsCount: 2 
replicasCount: 2 


.com/vl" 


Г\ HighLoad 
HL арка dd 


apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: "shardl-repll" 
layout: 
shardsCount: 1 
replicasCount: 2 
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apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: "shardl-repll" 
layout: 
shardsCount: 2 
replicasCount: 2 
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Что будем делать? 


ТА HighLoad 
(HL) м dd 


apiVersion: vl 
kind: PersistentVolume 
spec: 
persistentVolumeReclaimPolicy: Retain 


И еще немного кода вокруг зтого 
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Restore 


apiVersion: vl 
kind: PersistentVolume 
spec: 
awsElasticBlockStore: 
fsType: ext4 
volumeID: aws://eu-north-la/vol-0705ea080f82a429c 
capacity: 
storage: 7Gi 
claimRef: 
apiVersion: vl 
kind: PersistentVolumeClaim 
name: pvc-2 
namespace: dev 
resourceVersion: "52430830" 
uid: 2541da41-e996-4757-8£21-cb4a0£23de4b 
persistentVolumeReclaimPolicy: Retain A 
storageClassName: kops-ssd-1-17 ) нето 
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A для упрощения лучше сразу сделать 
правильный StorageClass 
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apiVersion: storage.k8s.io/vl 
kind: StorageClass 


metadata: 

name: kops-ssd-1-17-retain 
parameters: 

encrypted: "true" 

type: gp2 


provisioner: kubernetes.io/aws-ebs 
reclaimPolicy: Retain 
volumeBindingMode: WaitForFirstConsumer 
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apiVersion: "clickhouse.altinity.com/v1" 
kind: "ClickHouseInstallation" 
metadata: 
name: "simple-02" 
spec: 
configuration: 
clusters: 
- name: clusterl 
- name: cluster2 
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Восстановление данных: 


Возможно в полном объеме 
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Level: 
MOAR PERFORMANCE!!! 
Local storage 


Описание: 
Иногда пропадают куски данных 


Запрос: 
Верните все назад! 
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Persistent Storage — local storage 
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Восстановление данных: 
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Level: 
MOAR PERFORMANCE!!! 
Local storage 


Описание: 
Иногда пропадают куски данных 


Запрос: 
Верните все назад! dae 
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Читаем документацию 


for single node testing only; WILL 
NOT WORK in a multi-node cluster; 
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Persistent Storage — local storage 
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Pod 


Node 1 
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Что происходит при cMeHe Node, 
Ha которой исполняется Pod 
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Persistent Storage — local storage 
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Читаем документацию 


WILL NOT WORK in а multi-node 
cluster; 
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Что будем делать? 
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Affinity è anti-affinity 


Разложить Pods 
по нужным Nodes 


Example from operator 
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- name: clickhouse-per-host-on-servers-with-ssd 
spec: 


"hostname" 


affinity: 


# Specify Pod affinity to nodes with specified properties (hosttype=ch-ssd) 


nodeAffinity: 
requiredDuringSchedulingIgnoredDuringExecution: 
nodeSelectorTerms: 
- matchExpressions: 
- key: “hosttype" 
operator: In 
values: 
- “ch-ssd" 
# Specify Pod anti-affinity to Pods with the same label 


podAntiAffinity: 
reguiredDuringSchedulinglgnoredDuringExecution: 
- labelSelector: 
matchExpressions: 
- key: "clickhouse.altinity.com/app" 
operator: In 
values: 
- "chop" 
topologyKey: "kubernetes.io/hostname" 


"/app" on the same 


НО) Highload+ 


Весна 2021 


- name: clickhouse-per-host-on-servers-with-ssd 
spec: 
affinity: 


# Specify Pod affinity to nodes with specified properties (hosttype=ch-ssd) 


nodeAffinity: 
requiredDuringSchedulingIgnoredDuringExecution: 
nodeSelectorTerms: 
- matchExpressions: 
- key: “hosttype" 
operator: In 
values: 
- “ch-ssd" 
# Specify Pod anti-affinity to Pods with the same label 
"hostname" 
podAntiAffinity: 
requiredDuringSchedulingIgnoredDuringExecution: 
- labelSelector: 
matchExpressions: 
- key: "clickhouse.altinity.com/app" 
operator: In 
values: 
- "chop" 
topologyKey: "kubernetes.io/hostname" 


"/app" on the same 
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- name: clickhouse-per-host-on-servers-with-ssd 
spec: 
affinity: 


# Specify Pod affinity to nodes with specified properties (hosttype=ch-ssd) 


nodeAffinity: 
requiredDuringSchedulingIgnoredDuringExecution: 
nodeSelectorTerms: 
- matchExpressions: 
- key: “hosttype" 
operator: In 
values: 
- “ch-ssd" 
# Specify Pod anti-affinity to Pods with the same label 
"hostname" 
podAntiAffinity: 
requiredDuringSchedulingIgnoredDuringExecution: 
- labelSelector: 
matchExpressions: 
- key: "clickhouse.altinity.com/app" 
operator: In 
values: 
- "chop" 
topologyKey: "kubernetes.io/hostname" 


"/app" on the same 
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Docker 


K8s node 


Docker 


K8s node 


Docker 
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Указать 
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- name: clickhouse-per-host-on-servers-with-ssd 
spec: 
volumes: 
# Specify volume as path on local filesystem аз а 
directory which will be created, if need be 
- name: local-path 
hostPath: 
path: /mnt/podvolume 
type: DirectoryOrCreate 
containers: 
- name: clickhouse-pod 
image: yandex/clickhouse-server:20.7 
volumeMounts: 
# Specify reference to volume on local filesystem 
- name: local-path 
mountPath: /var/lib/clickhouse 
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Восстановление данных: 


Возможно в полном объеме 
Они и не терялись 
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Развитие — Local Volume 
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Persistent Storage — local storage 


Local - without manually scheduling . provisioning 
Pods to nodes, as the system is 

aware of the volume's node 

constraints 


K8s node 


Stateful Set 


Pod 
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Как так получается, что 
system is aware of the volumes node constraints? 
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Å вот так 


apiVersion: vl 
kind: PersistentVolume 
metadata: 
name: local-pv 
spec: 
local: 
path: /mnt/disks/ssdl 
nodeAffinity: 
required: 
nodeSelectorTerms: 
- matchExpressions: 
- key: kubernetes.io/hostname 
operator: In 
values: 
- nodel 
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Читаем документацию 


Local volumes can only Бе изей аза 
statically created PersistentVolume. 
Dynamic provisioning is not 
supported. 
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Что будем делать? 
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Нужен Provisioner! 
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Часть 2. Automation 
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Provisioner 


Independent program that follow a 
specification defined by Kubernetes. 
Authors of external provisioners 
have full discretion over where their 
code lives, how the provisioner is 
shipped, how it needs to be run 
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Static provisioning 
Dynamic provisioning 
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Автоматизация Static Provisioning 


Provisioner 
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Автоматизация Static Provisioning 


Node 


Mount point 1 


Mount point 2 


Автоматизация Static Provisioning 


x Mount point 1 


| x Mount point 2 
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Автоматизация Static Provisioning 


Mount point 1 


Mount point 2. 


Автоматизация Static Provisioning 


Автоматизация Static Provisioning 


Mount point 1 


. Mount point 2 


Автоматизация Static Provisioning 


apiVersion: storage.k8s.io/vl 
kind: StorageClass 
metadata: 

name: fast-disks 
provisioner: kubernetes.io/no-provisioner 
volumeBindingMode: WaitForFirstConsumer 
reclaimPolicy: Delete 


https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner 
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Читаем документацию 


Local volumes can only Бе изей аза 
statically created PersistentVolume. 
Dynamic provisioning is not 
supported. 


C HighLoad 
(HL) FREI dd 


Вроде бы нельзя. 
Но очень 
хочется. 
Надо пробовать. 
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Dynamic Provisioning 


Provisioner 
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Dynamic Provisioning 


Node 


Mount point 1 


Mount point 2 


Dynamic Provisioning 


Mount point 1 


Mount point 2 


Dynamic Provisioning 


Mount point 2. 


Dynamic Provisioning 
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Dynamic Provisioning 


Mount point 1 


Mount point 2 
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Dynamic Provisioning 


Mount point 1 


Mount point 2 
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Dynamic Provisioning 


apiVersion: vl 


apiVersion: storage.k8s.io/vl kind: PersistentVolumeClaim 
kind: StorageClass metadata: 
metadata: name: dynamic-local-pvc 
name: dynamic spec: 
provisioner: dynamic-local-provisioner accessModes: 
volumeBindingMode: WaitForFirstConsumer - ReadWriteOnce 
reclaimPolicy: Delete storageClassName: dynamic 
resources: 
requests: 


storage: 128Mi 


https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner 


https://github.com/rancher/local-path-provisioner/ 
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lib 
A library for writing external provisioners 


https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner 
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Persistent Data в Kubernetes может быть 
даже с локальными дисками 
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ClickHouse Operator Github Project: 


https://github.com/Altinity/clickhouse-operator 


Issues & Pull Requests on Github 
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